THE SUBSIDIZED AND TRANSITIONAL EMPLOYMENT DEMONSTRATION 


MDRC Working Paper 


COSTTANRAIS 
OF THE 
STEP FORWARD 


NOG NANI, 


OPRE Report 2019-109 


November 2019 


The Subsidized and Transitional Employment Demonstration: 
Cost Analysis of the STEP Forward Program 


OPRE Report 2019-109 
November 2019 


Author: Riley Webster, MEF Associates 


Submitted to: 


Girley Wright, Project Officer 
Office of Planning, Research, and Evaluation 
Administration for Children and Families 
U.S. Department of Health and Human Services 


Project Director: Dan Bloom 
MDRC 
200 Vesey Street 
New York, NY 10281 


Contract Number: HHSP 233-2010-0029YC 
This report is in the public domain. Permission to reproduce is not necessary. 
Suggested citation: Webster, Riley. 2019. Cost Analysis of the STEP Forward Program. OPRE 
Report 2019-109. Washington, DC: Office of Planning, Research, and Evaluation, 
Administration for Children and Families, U.S. Department of Health and Human Services. 
Disclaimer: The views expressed in this publication do not necessarily reflect the views or 
policies of the Office of Planning, Research, and Evaluation, the Administration for Children and 


Families, or the U.S. Department of Health and Human Services. 


This report and other reports sponsored by the Office of Planning, Research, and Evaluation are 
available at www.acf.hhs.gov/opre. 


a A ae Like OPRE on Facebook Follow OPRE on Twitter 
fat! E-Newsletter __ facebook.com/OPRE.ACF @OPRE ACF 


=OPRE mdrec 


BUILDING KNOWLEDGE 
TO IMPROVE SOCIAL POLICY 
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Health and Human Services (HHS), funded by HHS under a competitive award, Contract No. 
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Foundation, The Joyce Foundation, The Kresge Foundation, and Sandler Foundation. 
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tion, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth 
and Frank Newman Charitable Foundation, The New York Times Company Foundation, Jan 
Nicholson, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The 
Stupski Family Fund, as well as other individual contributors. 


For information about MDRC and copies of our publications, see our website: www.madrc.org. 


Overview 


Introduction 


STEP Forward was a voluntary program in San Francisco that aimed to connect low-income 
job seekers to the labor market by using public funds to temporarily subsidize individuals’ 
wages, known as subsidized employment, with the goal of ultimately increasing permanent 
unsubsidized employment among this population. The program was operated by the Human 
Services Agency of San Francisco (HSA) under the umbrella of JOBSNOW!, HSA’s broader 
subsidized employment initiative. STEP Forward offered job seekers opportunities to inter- 
view for jobs with private sector employers at weekly job fairs and offered employers tempo- 
rary wage subsidies if they hired disadvantaged job seekers whom they might not otherwise 
hire. A diverse group of low-income job seekers enrolled in the program, the vast majority of 
whom were either CalWORKs (California’s Temporary Assistance for Needy Families pro- 
gram) clients, individuals who had exhausted their unemployment insurance benefits, or 
CalFresh (California’s Supplemental Nutrition Assistance Program) recipients. 


To test the effectiveness of this strategy and to learn about its costs, the Administration for 
Children and Families (ACF) in the U.S. Department of Health and Human Services funded 
a random assignment evaluation of STEP Forward, in which individuals were randomly as- 
signed to a program group that had access to STEP Forward services or to a control group 
that did not have access to STEP Forward services but could receive other services in the 
community. A total of 837 adults were enrolled into the STEP Forward study between No- 
vember 2012 and March 2015. In the first year after random assignment when program 
group members who were placed in subsidized employment were still receiving subsidized 
wages, program group members were more likely than control group members to have been 
employed and had higher average earnings. Impacts on earnings continued into the fourth 
year, well after the subsidies had ended. 


This paper presents the one-year costs of STEP Forward, including program services such 
as job-readiness services and the subsidized portion of wages provided to program group 
members. It also presents the costs of non-STEP Forward services available to both pro- 
gram group and control group members, which included education, training, and job skills 
services. This evaluation is part of a larger demonstration called the Subsidized and Transi- 
tional Employment Demonstration (STED). In addition to STEP Forward, STED includes 
random assignment evaluations of six other subsidized employment strategies around the 
country. MDRC, a nonprofit, nonpartisan research organization, is leading the project under 
a contract with ACF along with its partner, MEF Associates. Longer-term findings from all 
STED random assignment evaluations are included in a separate synthesis report, while 
earlier findings discussing the implementation of STEP Forward and early impacts are in a 
separate interim report. 


Purpose 


The purpose of the cost study is to determine what it cost to provide STEP Forward services 
to a single program group member once STEP Forward had reached a steady state of op- 
eration. The analysis estimates the costs per sample member in two categories: STEP For- 
ward program costs and non-STEP Forward costs, which include remedial education, post- 
secondary education, vocational training, and job skills services. 


Key Findings and Highlights 


e The cost of STEP Forward program services averaged $2,646 per program group mem- 
ber. Program group members received an average of $1,038 in subsidized wages. 


e The cost of non-STEP Forward services, which both groups received, averaged $1,751 
per program group member and $2,355 per control group member. 


e The net cost, the difference between the total program group costs and the total control 
group costs, averaged $3,080 per program group member. 


Methods 


The cost of STEP Forward was assessed using program expenditure reports, program sur- 
vey data, and public data sources. The cost of STEP Forward program services were esti- 
mated by summing direct program expenditures from July 2013 through June 2014 and 
dividing by the number of program group members enrolled during that period. To estimate 
the cost of non-STEP Forward program services, the research team first determined the unit 
cost, or the cost of serving one person for a specific unit of time (for example, one month). 
Multiplying the unit cost by the average length of time sample members received each ser- 
vice gives the average cost incurred per sample member. 
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Introduction 


Securing unsubsidized employment can be difficult for low-income job seekers in a com- 
petitive labor market that is increasingly dominated by highly skilled technical and pro- 
fessional employment. This is especially true in San Francisco, whose tech boom has 
not benefited workers who are less educated or who lack the higher skills required by 
the jobs that have driven the economy’s recent growth. Securing stable employment is 
a must for lower-income workers in both San Francisco and the United States at large. 


STEP Forward was a voluntary program in San Francisco that aimed to con- 
nect low-income job seekers to the labor market by using public funds to temporarily 
subsidize individuals’ wages, known as subsidized employment, with the goal of ulti- 
mately increasing permanent unsubsidized employment among this population. The 
program was operated by the Human Services Agency of San Francisco (HSA) under 
the umbrella of JOBSNOW!, HSA’s broader subsidized employment initiative. STEP 
Forward offered job seekers opportunities to interview for jobs with private sector em- 
ployers at weekly job fairs and offered employers temporary wage subsidies if they 
hired disadvantaged job seekers whom they might not otherwise hire. A diverse group 
of low-income job seekers enrolled in the program, the vast majority of whom were 
either CalWORKs (California’s Temporary Assistance for Needy Families program) 
clients, individuals who had exhausted their unemployment insurance benefits, or 
CalFresh (California’s Supplemental Nutrition Assistance Program) recipients. 


To test the effectiveness of this strategy, the Administration for Children and 
Families (ACF) in the U.S. Department of Health and Human Services funded a random 
assignment evaluation of STEP Forward, in which individuals were randomly assigned 
to a program group that had access to STEP Forward services or to a control group that 
did not have access to STEP Forward services but could receive other services in the 
community.’ A total of 837 adults were enrolled into the STEP Forward study between 
November 2012 and March 2015. 


This evaluation is part of a larger demonstration called the Subsidized and Tran- 
sitional Employment Demonstration (STED). In addition to STEP Forward, STED in- 
cludes random assignment evaluations of six other subsidized employment strategies 
around the country. MDRC, a nonprofit, nonpartisan research organization, is leading 
the project under a contract with ACF along with its partner, MEF Associates. Longer- 
term findings from all STED random assignment evaluations are included in a separate 
synthesis report,? while earlier findings discussing the implementation of STEP Forward 
and early impacts are in a separate interim report.* 


‘Individuals who were randomly assigned to the control group received a list of other services, 
such as assistance with housing, transportation, and job search, which they could access on their 
own. 

2Cummings (forthcoming). 

3Walter, Navarro, Anderson, and Tso (2017). 


This paper presents the one-year costs of STEP Forward, including the subsi- 
dized portion of wages provided to program group members, and the cost of providing 
education, training, and job skills services. It begins with a description of the services 
offered through the program and a discussion of the program’s impacts, followed by a 
review of the data sources and a description of the methodology used to estimate the 
costs. It then presents an estimate of the cost of services for the program group and the 
difference in the cost of services provided to program group members relative to the 
control group. 


STEP Forward Program Services 


The STEP Forward program was designed to operate similarly to a staffing agency, 
offering job seekers opportunities to interview for jobs with private sector employers at 
weekly job fairs. The program offered employers temporary wage subsidies to encour- 
age them to try out job seekers whom they might not otherwise hire, with the goal that 
they might ultimately hire these workers into permanent unsubsidized employment. 


Once enrolled in the voluntary program, participants met with a case manager 
who assessed if they were ready to interview for jobs based on whether they had an 
updated résumé, appropriate interview attire, and access to the necessary child care 
and transportation. If the participant met these requirements, the case manager re- 
viewed the available job openings that employers had listed with the program and com- 
pleted a worksheet that ranked how well the participant fit a particular job, selecting the 
highest ranked participants to interview with that position’s employer (as long as those 
participants expressed interest in the position). 


In order for employers to qualify for subsidies, the jobs they offered had to be at 
least 25 hours per week and pay the prevailing wage for the position. HSA provided the 
subsidy reimbursement, which varied based on the position’s hourly wage but had a 
maximum of $1,000 per month for up to five months. 


For those participants who did not meet the requirements, the case manager 
worked with them to help prepare them for the job market by reviewing their résumés, 
conducting practice interviews, or referring them to outside providers for services and 
resources HSA did not offer. Once participants successfully obtained a subsidized job, 
HSA considered them to have completed the program and HSA staff did not initiate post- 
placement contact. 


STEP Forward’s Impacts 


To assess to what degree STEP Forward affected the kinds of services the program 
group received and to what extent these services improved their employment outcomes, 
the research team looked at employment and earnings data from the National Directory 
of New Hires one, two, three, and four years after random assignment and survey data 


from one year after random assignment for both the program group and the control 
group.* Some of STEP Forward’s impacts on service receipt and employment out- 
comes, shown in Table 1, are discussed below, but a more complete discussion can be 
found in the interim and final synthesis reports. 


e The program greatly increased receipt of employment services, 
which was unsurprising given that most of the program group re- 
ceived job-readiness services through STEP Forward, while con- 
trol group members could receive such services only through 
other programs and services they found in the community. 


While over half of the control group reported receiving help with finding or keep- 
ing a job, likely through other programs and services they found in the community, nearly 
94 percent of the program group received these services. These services included help 
with job searches, job referrals, developing a résumé, filling out job applications, and 
preparing for job interviews, among other activities. STEP Forward offered many of 
these services to program group members in the form of job-readiness activities. Only 
a quarter of program group members were interviewed and hired through the program. 


e In the first year after random assignment, program group mem- 
bers were more likely than control group members to have been 
employed and had higher average earnings. Impacts on earnings 
continued into the fourth year. 


Nearly 70 percent of the control group worked in the year following random as- 
signment, according to administrative records. However, three-fourths of program group 
members were employed in the year following random assignment, resulting in an im- 
pact on employment of 6 percentage points. Program group members also earned ap- 
proximately $1,600 more (including the subsidy amount) than control group members, 
on average, in the year following random assignment. Program group members contin- 
ued to earn more even after the subsidies ended in the second, third, and fourth years 
following random assignment, earning $2,142, $2,181, and $2,941 more than control 
group members, respectively. 


Methodology and Data Sources 


This analysis estimates the cost of providing STEP Forward services to a single program 
group member once STEP Forward had reached a steady state of operation. As such, 
the analysis uses HSA expenditure data from July 2013 to June 2014. By July 2013, the 


4Unless otherwise indicated, all impacts discussed in this report are statistically significant, with 
p-values less than 0.10 — meaning that there is less than a 10 percent chance that the observed 
impacts were not a result of the program. 


Table 1 


STEP Forward Impacts on Service Receipt and Employment Outcomes 


Outcome 


Year 1 (self-reported outcomes) (%) 
Received help related to finding or keeping a job 
Participated in education and training 
ESL, ABE, or high school diploma or equivalency classes 
Postsecondary education leading to a degree 
Vocational training 


Sample size (total = 556) 


Year 1 (administrative outcomes) 
Employment (%) 

Participated in STED subsidized employment 
Total earnings ($) 

Subsidized earnings 


Year 2 (administrative outcomes) 
Employment (%) 
Total earnings ($) 


Year 3 (administrative outcomes) 
Employment (%) 
Total earnings ($) 


Year 4 (administrative outcomes) 
Employment (%) 
Total earnings ($) 


Sample size (total = 811) 


Program Control 


Group 


93.8 
32.1 
6.2 
9.2 
22.1 


286 


75.7 
24.1 
8,521 
1,016 


76.7 
14,413 


76.4 
17,433 


75.2 
19,521 


414 


Group 


57.7 
31.5 

6.4 
13.4 
16.2 


270 


69.7 
N/A 
6,912 
N/A 


73.5 
12,271 


74.5 
15,252 


71.9 
16,580 


397 


Difference 
(Impact) 


36.1 *** 
0.6 
-0.1 
-4.2 
5.9 * 


6.0 * 


1,610 *** 


3.2 
2,142 * 


2.0 
2,181 * 


3.3 
2,941 ** 


90 Percent 
Confidence 
Interval 


[30.6, 41.5] 
[-6.0, 7.2] 
[-3.3, 3.1] 
[-8.8, 0.3] 
0.3, 11.5] 


[1.3, 10.7] 


[591, 2,628] 


-1.5, 7.9] 
[473, 3,811] 


[-2.8, 6.7] 
[174, 4,189] 


[-1.6, 8.2] 
(745, 5,137] 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, STEP 
Forward subsidy payment records, and responses to the 12-month survey. 


NOTES: ESL = English as a Second Language; ABE = Adult Basic Education. 
Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


program was operating at full capacity, so this time period was chosen to avoid including 
costs associated with start-up activities. The analysis also estimates the costs of non- 
STEP Forward services available in the community for both program and control group 
members. 


This analysis estimates the costs per sample member in two categories (shown 
in Figure 1): STEP Forward costs and non-STEP Forward costs. To determine the cost 
per sample member for each service within the latter category, the research team first 
determined the unit cost, or the cost of serving one person for a specific unit of time (for 
example, one month). Multiplying the unit cost by the average length of time sample 
members received each service gives the average cost incurred per sample member. 
All costs have been adjusted to 2016 dollars for this analysis. 


STEP Forward Costs 


HSA requires staff to itemize hours worked by program, including STEP For- 
ward, every quarter. The cost of the STEP Forward program was calculated by allocat- 
ing a portion of staff salaries and administrative costs based on the number of hours 
staff recorded spending on STEP Forward activities. The allocated expenditures were 
divided by the number of program group members randomly assigned to the program 
between July 2013 and June 2014 to calculate a cost per program group member. Sub- 
sidized wages for the program group members came from payroll records. 


Non-STEP Forward Costs 


Sample members could access a variety of services from outside providers in- 
cluding remedial education training, postsecondary education and vocational training, 
and job skills training. Data on sample member’s participation in non-STEP Forward 
services came from a participant survey administered 12 months after random assign- 
ment. The survey asked respondents to estimate the number of weeks they spent in 
various activities, shown in Table 2. Weekly unit costs (the estimated costs of serving 
one person for one week) were constructed for each of these activities based on infor- 
mation from published sources, also shown in Table 2. 


The costs of remedial education were calculated for California from the U.S. 
Department of Education’s Office of Career, Technical, and Adult Education’s National 
Reporting System and included participation in English as a Second Language, adult 
basic education classes, classes to prepare for a high school diploma, and high school 
equivalency classes, including those leading to a General Educational Development 
certificate. 


The research team assumed sample members received postsecondary and vo- 
cational training services from the public community college system in San Francisco 
and estimated the costs of these services from the U.S. Department of Education’s Na- 
tional Center for Education Statistics’ Integrated Postsecondary Education Data System. 


Figure 1 


STEP Forward Cost Components 


Program Group 


STEP Forward Costs 
STEP Forward services 
¢ Job-readiness services 


¢ Job skills referrals 
* Job development 


STEP Forward expenses 
* Subsidized wages 


Non-STEP Forward Costs 

* Education and training 

o Remedial education 

o Postsecondary 
education 


(@Zo}n) (xe) MEI ce}0] 0) 


Non-STEP Forward Costs 
Education and training 
o Remedial education 
o Postsecondary 


education 
o Vocational training 
¢ Job skills services 


o Vocational training 
¢ Job skills services 


Total Costs per Control 
Group Member 
(D=E) 


Total Costs per Program Group 
Member 
(C=A+B) 


Differences = 
Net cost per program 
group member (F = C - E) 


Job skills services included receiving help with job searching, job referrals, de- 
veloping a résumé, filling out job applications, preparing for job interviews, job-readiness 
training, and planning for future career or educational goals. The research team did not 
know where sample group members received these services, so this analysis uses the 
cost of a class or workshop from the Workforce Investment Act Adult and Dislocated 
Worker Programs Gold Standard Evaluation. 


Table 2 
Unit Costs and Participation Information 


Length of Participation (in Weeks) 


Component Weekly Cost ($) Program Group Control Group 

Remedial education 113 1.2 0.9 
Postsecondary education 245 3.9 5.4 
Vocational training 228 2.2 2.2 
Job skills 58 2.4 7.9 


SOURCES: Calculations are based on data from U.S. Department of Education's 
National Center for Education Statistics’ Integrated Postsecondary Education Data 
System; U.S. Department of Education’s Office of Career, Technical, and Adult Education 
National Reporting System; Mastri and McCutcheon (2015); and responses to the 12- 
month survey. 


Costs 


Average costs are divided into two categories — STEP Forward services and non-STEP 
Forward services — for the program and control groups. (See Table 3.) 


STEP Forward Costs 


The monthly cost of STEP Forward program services was $2,646 per program 
group member. Program group members received $1,038 in subsidized wages on av- 
erage, for a total cost of $3,684 per program group member. 


Non-STEP Forward Costs 


Although program and control group members were equally likely to participate 
in remedial education, among those who participated, program group members spent 
more weeks in remedial education than did control group members. The per-person 
costs of these services averaged $102 for the control group, compared with $139 for the 
program group. Program group members were less likely to participate in postsecondary 
education, and the per-person costs of these services averaged $1,290 for the control 
group and $963 for the program group. Control group members and program group 
members spent a similar number of weeks on average in vocational training, with these 
costs averaging $506 for the control group and $510 for the program group. Program 
group members were more likely to receive help related to finding or keeping a job. The 
majority of program group members received this help as part of STEP Forward, and 
the cost is already captured as part of the STEP Forward cost. Control group members 


Table 3 


Estimated Net One-Year Cost per Program Group Member 
(in 2016 Dollars) 


Gross Costs Net Costs per 

Component ($) Program Group _—_ Control Group Program Group Member 
STEP Forward costs 

Staff services 2,646 0 2,646 
Subsidized wages 1,038 0 1,038 
Total STEP Forward costs 3,684 0 3,684 
Non-Step Forward costs 

Remedial education 139 102 37 
Postsecondary education 963 1,290 -327 
Vocational training 510 506 4 
Job skills training 139 457 -318 
Total non-STEP Forward costs 1,751 2,355 -604 
Total Costs 5,435 2,355 3,080 


SOURCES: Calculations for STEP Forward costs based on fiscal data from the Human Services 
Agency of San Francisco, participation data and wages from the JOBSNOW! management information 
system data, and subsidy payment records. Calculations for non-STEP Forward costs based on data 
from U.S. Department of Education's National Center for Education Statistics’ Integrated 
Postsecondary Education Data System; U.S. Department of Education’s Office of Career, Technical, 
and Adult Education National Reporting System; Mastri and McCutcheon (2015); and responses to the 
12-month survey. 


would have received this help from other community providers. The per-person costs of 
these services averaged $457 for the control group and $139 for the program group.® 


Net Costs 


The net cost is the difference between all program group costs and all control 
group costs and reflects what was spent on program group members over and above 
what was spent on control group members. The per-person net cost averaged $3,080, 
which is the lowest per-person net cost of any STED program. This low cost combined 
with the large impacts on earnings through three years after random assignment 
means STEP Forward was likely cost-beneficial from society’s perspective. For a full 
discussion of costs across different subsidized employment program types, see the 
final synthesis report. 


‘The cost of these services for program group members does not include the costs of job skill 
services provided through STEP Forward. 
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